Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When doing a retry, throw some randomness into the mix #140

Merged
merged 1 commit into from
Mar 7, 2024

Conversation

Luc45
Copy link
Member

@Luc45 Luc45 commented Mar 7, 2024

Minor improvement to the retry logic, especially useful for our parallel tests.

What this fixes:

  • We dispatch everything in parallel
  • We hit a 429
  • Server ask us to retry after 10 seconds
  • Then we retry every request that hit a 429 after 10 seconds, which can incur into another 429

This PR throws some randomness into the mix, waiting the specified Retry-After specified by the server, and adding between 0 and 5 seconds of additional wait, to avoid all retries being done in the same time.

In the current behavior, we can see that several requests hit 429 at 12:34:17, and retried at 12:34:27.

image

@Luc45 Luc45 requested a review from a team March 7, 2024 12:55
@Luc45 Luc45 self-assigned this Mar 7, 2024
@Luc45
Copy link
Member Author

Luc45 commented Mar 7, 2024

This is a pretty straightforward PR and I'll be moving forward with the merge to reduce code review churn.

@Luc45 Luc45 merged commit 4e6b883 into trunk Mar 7, 2024
1 check passed
@Luc45 Luc45 deleted the 24-03/retry-with-variance branch March 7, 2024 13:03
@Luc45
Copy link
Member Author

Luc45 commented Mar 7, 2024

We can see that with this PR the retries are a little bit more spaced out between them
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant